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ABSTRACT 

On September 30, 1993, Far West Laboratory (FWL) and 
the California Assessment Collaborative (CAC) sponsored a 
cost/benefit forum to examine cost-analys is issues associated with 
the development and implementation of instructionally sound 
assessment. This report highlights the central ideas addressed at the 
forum, incorporates the audience participation and feedback, and 
casts the discussion in the context of local assessment reforms faced 
by California schools. Following a description of the forum 
proceedings, a summary of the major conceptual and methodological 
issues is presented. These issues included the infusion and 
allocation of limited resources, the distinction between expenditures 
and costs, types of benefits (benefits of the reform and those 
foregone), and factors influencing cost estimates (efficiency, 
alignment, and substitution). It is argued that the range of cost 
must be established in order to place some realistic boundaries on 
discussions of resources required for successful, local assessment 
reform. Educators and policy makers need to develop more 
sophisticated models of costs and benefits beyond the expenditure 
analysis most typically presented. Such models must include analyses 
of alternatives both in terms of expected benefits from the chosen 
option as well as the benefits foregone from those rejected. They 
should also include a range of scenarios that reflect high and low 
levels of substitution and efficiency. Rarely will reforms be paid 
fully by incorporating substitution methods. Consequently, policy 
makers must provide evidence of efficiency along with the arguments 
of alignment that are typically proffered when new programs are 
introduced. (LMI) 
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Cost Analysis and Instructionally Sound Assessment Practices: 
Issues for Local Exploration 
Peter Ryan 


Far West Laboratory for Educational Research and Development 


On September 30, 1993, Far West Laboratory (FWI) and the California Assessment Collaborative (CAC) 
sponsored a cost/benefit forum to help illuminate issues surrounding cost analysis associated with 
development ‘ad implementation of instructionally sound assessment. One of the charges of the CAC is tc 
collect and analyze preliminary cost data related to implementing new assessments and to contrast these 
with the costs of more traditional testing options. Recognizing that the CAC focus on assessment 
development in local schools and districts represents only one component of systemic assessment reform 
in California, practitioners and researchers affiliated with a other state and local reforms were invited 
to participate. Forum participants included CAC consortium leaders and project leaders from individual 
districts, representatives from the California Department of Education, California State University, 
California Career-Technical Assessment Project, Far West Laboratory, and key policy groups, including 
the Office of the Governor. The forum served as an opportunity to discuss new developments in cost 
analysis associated with assessment reforms and to place this information in the context of systemic 
reform in California. This report highlights the central ideas addressed at the forum, incorporates the 
audience participation and feedback, and casts the discussion in the context of local assessment reforms 
faced by California schools. Following a description of the event, a summary of the major conceptual and 


methodological issues discussed at the forum is presented. 


The Cost/Benefit Forum 


The n speaker for the forum was Dr. David Monk, Professor, Cornell University. In an interactive 
discussion format, Dr. Monk presented a conceptual overview of cost analysis as a tool for education 
reform. He addressed recent improvements in the conceptualization of costs as well as methodological 
advances. This cast a positive light on cost analysis in education, but also highlighted many pitfalls that 
can result in misleading information for policy makers and stakeholders engaged in reform efforts. The 
presentation was buttressed by examples and cost estimates from Dr. Monks work conceptualizing and 


generating preliminary cost information for the New Standards Project. 


Following Dr. Monk's exposition, Dr. Kate Jamentz, director of the CAC, introduced the conceptual map 
illustrating patterns of influence in instructionally sound assessment. This map forwards five essential 
dimensions of educational of practice related to instructionally sound assessment: (a) articulating content 
standards, (b) developing meaningful, fair assessments, (c) building teacher capacity to use assessment to 
improve instruction, (d) building student capacity to use assessment to improve learning, and (e) 
determining and monitoring the consequences of the assessment. Each dimension suggests variables that 
influence whether an assessment proves instructionally sound. Dr. Jamentz then described the CAC 
framework for the analysis of benefits and costs of performance assessment and illustrated linkages with 
the CAC conceptual map. Significant benefits appear possible for students, educators, and communities 


from instructionally sound assessment practices. However, even the most optimistic scenarios imply 


substantial costs for implementing and sustaining the reforms. 


This part of the Forum was followed by an extended question and answer session on specific applications 
of cost analyses in state and local contexts. Much of the discussion focused on the next steps for 
conceptualizing and gathering initial cost estimates for local projects. Specifically, discussions 
emphasized the need to clearly articulate and gather cost information related to "challenges that imply 
costs" and costs for teachers, students, and communities that typically do not appear in expenditure data. 
For example, this might include costs associated with changing parent and community expectations, 
building teacher content knowledge and instructional expertise, and addressing inequalities in 
Opportunities to learn. This would be a significant contribution because at this initial stage of systemic 
assessment reform not much is understood about the costs born to realize the projected benefits. When 
conceptualizing and collecting cost information, a strong suit for the CAC is the availability of multiple 
projects encompassing different, but related assessment reforms. This affords the opportunity to 
investigate costs related to a single dimension across projects or multiple dimensions within a single 
project. Delineation of costs enhances our ability to place realistic parameters on the resources required to 


initialize and sustain current assessment reforms. 


Summary of Major Conceptual Issues 


Infusion and Allocation of Limited Resources 


There is little disagreement that realizing the new educational reforms will require either the substantial 
infusion of new resources or the redirection of existing ones. Given that educational systems have finite 
resources to draw from, reforms within the system compete for available funds. Thus, educators and 
policy makers are continually seeking the most effective methods for producing a given set of desired 


outcomes. Cost analysis can inform policy makers by providing crucial information that serves to either 
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direct resources toward or away from certain practices. However, without careful consic eration of the 


conceptual and methodclogical complexities of cost analysis, the results can be misleading and do more 


harm thaa good.. 


Distinguishing Expenditures and Costs 


* 


Much of what is done under the auspices of cost analysis actually is expenditure analysis. For example, in 
traditional mathematics assessment, when estimating resources required to purchase an existing testing 
program, a district might consider the costs of testing materials, scoring services, reporting and 
dissemination, and may even include costs of teacher time devoted to the testing program. This might be 
compared to the costs associated with a district-developed, open-ended mathematics assessment. Among 
estimated resources for this type of assessment would be expenses for teacher release time to develop and 
pilot items, production of assessment materials, teacher time for administration, teacher time for scoring 
the items, and reporting and disseminating of results. Based on this information, the district might 


conclude that the expenditures for these two assessment programs are quite different. While this is true, 


the district has not conducted a cost analysis. Cost functions are based on the assumption that the outcome 


or benefit side of the equation is constant. In the framework of cost analysis, costs cannot be divorced 
from the production or benefit side. In the example, the intended benefits and the actual benefits that 
accrue from the various assessment strategies are quite different. Here, summing-up and comparing 


expenditures might be misinterpreted as a full cost analysis. 


Expenditure analysis is often conducted because documentation of expenses is readily accessible by 


educators and policy makers. However, cost data are much more complex to conceptualize and collect and 


there are fewer individuals experienced in utilizing cost data. 


Types of Benefits 


Districts cannot estimate costs without dealing with the outcomes or benefits of the assessment. Two types 
of benefits often found useful for analyzing costs are: (1) the benefits the district obtains from the reform 


itself; and (2) the benefits foregone because resources are not directed to alternatives. 


Benefits of the reform. In our mathematics assessment example, the proposed benefits of the open-ended 
mathematics assessment might include: to enhance student and teacher capacity to engage in complex 
tasks; to drive teaching and learning toward an enriched curriculum; to raise expectations of student and 
teacher performance; to increase communication about the content of the new state frameworks among 
parents and community; and to reward students, teachers, and schools that meet high performance 
standards. Because the proposed benefits of the assessment reform are quite varied and substantial, 
identifying relevant costs quickly becomes complex. The challenge for the district is to know how to 


allocate resources to accomplish the goals and related benefits of the assessment. 


Benefits foregone. Benefits foregone are related to the notion of opportunity costs. When resources are 
allocated to a given assessment, the benefits associated with the alternative uses of those resources bear on 
the estimation of costs. Although there are many alternative uses, costs of benefits foregone are usually 


associated with the best alternative use. Identifying the best alternative use is not easy. It depends upon 
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what are considered viable and reasonable alternatives. In our example, reasonable alternatives to the 
open-ended mathematics assessment might be the use of a traditional assessment, the implementation of a 
published performance assessment, or the allocation of resources to curriculum development. Costing out 
the benefits foregone depends on how the district defines the reasonable range of alternative uses. If the 
range of alternative uses includes several attractive alternatives, the costs for the benefits foregone will be 


* 


relatively high. Alternately, it does not cost much to give up an unattractive alternative. 


Factors Influencing Cost Esti 


Efficiency, alignment, and substitution are three factors that can affect the costs of an assessment reform 


effort. 


Efficiency. New forms of assessment have been criticized for being relatively inefficient--they often 
consume large amounts of time to develop, administer, and score, in order to achieve their promised 
benefits. However, traditional forms of assessment may also be inefficient in the amount of resources 
expended for the outcomes produced. For example, if teachers spend time preparing and testing students 
using tests that are not closely aligned with current curricular reforms, then the resources expended to 
obtain estimates of student competence may have limited utility for students, teachers, and districts, and 
thus may be excessive. Efficiency becomes especially important when resources are limited and real 
alternatives exist. Furthermore, if efficiency of the assessment is low, then the cost of giving up the 
practice diminishes, all other factors being relatively equal. This argument holds true for both traditional 


and new forms of assessment. 


: 
| 


Alignment. Closely related to the efficiency of producing desired educational outcomes is the issue of 
alignment. If local resources are devoted to an assessment reform initiative, then traditional assessment 
often remains the most likely alternative. To the extent that the benefits yielded by the traditional 
assessment are aligned with state and local reform efforts, the costs for benefits foregone diminish. 
However, if the benefits of the traditional assessment are judged not to be closely aligned, then the cost of 
the beastie foregone by not investing in a new assessment increase. In the context of current, systemic 
reform efforts that promote high standards and an enriched curriculum, traditional assessments might be 
viewed as a practice that impedes reform. If the perceived misalignment between the assessment and 
curriculum framework is severe, it may drive a restricted curriculum, provide information of limited utility 


to teachers, and misinform the public about the performance of students. The point is that it costs less to 


give up a practice that is counter to current reform efforts, than one that is closely aligned. 


Substitution. The cost of implementing assessment reform also depends on the extent to which new 
resources must be marshaled to support the reform effort. If the costs of the assessment reform are 
absorbed because they substitute for existing practices, then implementation costs will be reduced. 
Proponents of new assessments see great potential for this to occur. For example, they envision that 
performance assessment will substitute for traditional assessment, blur the distinction between instruction 
and assessment, and blend with curriculum development. In addition, most will concede that new forms of 
assessment will require substantial investment in teacher professional development at both preservice and 
inservice levels. However, this investment may not be viewed as a complete add-on to existing programs. 
Most districts in California, for example, have a number of professional development days which are often 
allocated to a potpourri of workshops, conferences, and topics over the years. If a portion of these existing 


resources were allocated to the assessment reform effort, the substitution would reduce both start-up and 
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maintenance costs associated with the reform. If, on the other hand, the assessment reform is an add-on to 


existing practices, the resources necessary for successful implementation may be prohibitive. 


A Strategy for Formulating Preliminary Cost Estimates 


In conventional economic analyses, the dollar is the common denominator when estimating costs and 
valuing benefits. At this point, we are just beginning to identify the range and complexity of costs and 
benefits associated with assessment reforms. Furthermore, most current research and published 
information documents the costs and benefits of state and national reform efforts. The costs associated 
with these national and state level reforms often have little bearing on efforts at the district level. The 
reality is that most of the costs associated with assessment reform will be embedded in professional 
development efforts and implementations in schools and classrooms--areas in which districts will bear the 
costs. It would bode well for school districts engaged in assessment reform to investigate the possible 


range and magnitude of assessment reform costs. 


Just summing the dollar values of costs to attain certain benefits may misinform educators planning 
assessment reforms. Due to the recency of the reforms and uncertainties of the real costs, it may be too 
early to provide accurate point estimates of costs related to these new assessments. In this context the 
precise measurement and estimation of costs may be formidable. Consider for a moment the statistician in 
a similar position. Statisticians know that certain point estimates are loaded with error due to imprecision 
in conceptual understandings and measurement procedures. To provide more useful approximations, they 
employ interval estimates. These estimates establish upper and lower boundaries and acknowledge the 
imprecision in estimating a given quantity or effect. Analogously, when estimating assessment reform 
costs, educators might create a range of scenarios. These scenarios will reflect, for example, high and low 
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levels of substitution and efficiency which will directly affect the costs associated with realizing the 


benefits of the assessment reform. 


In an assessment reform that has high substitution and efficiency, many existing practices would be 
replaced by the rew assessment-- limiting any extra costs. In addition, in an efficient system, the attained 
benefits would be maximized for a given set of resources. For example, substitution may involve 
replacement of the traditional standardized test and a criterion-referenced assessment with the assessment 
reform. In addition, several teacher professional days could be allocated from less valued workshop topics 
to teacher involvement in development and scoring of the new assessment. In terms of efficiency, the 
teacher workshops on assessment development and scoring are well-attended and provide the necessary 
boost that a qualified teaching staff needs to implement new instruction and assessments in their 
classrooms. In th s scenario, modest infusion of new resources results in substantial benefits to students 


and teachers due to the assessment reform's high substitution, efficiency, and alignment. 


Alternately, a si:enario characterized by low substitution and low efficiency produces an assessment that is 


marked by mut:h higher costs to realize the intended benefits. In this case, the assessment reform might be 


an add-on to the traditional assessment practices and create a burdensome assessment situation, where the 


gathering of rew information requires additional hours of test-taking and scoring. The teacher 
development days marked for assessment reform are added io existing programs and not well-planned or 
received. This results in little or no classroom impact and low quality assessment information. Thus, in 
this example, substantial infusion of resources or high costs may be necessary to produce the intended 
benefits. The extra-testing burden and poor planning by the district also may make the foregone benefits 
more attractive than the new assessment reform itself. It is potentially very expensive to substitute an 
accepted practice, even one with limited perceived benefit, with a poorly planned, reform-based practice. 
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In the two scenarios presented above, the costs to realize the benefits of the assessment reforms would be 


very disparate. The reality is that most districts will fall somewhere in between the high and low cost 
scenarios. By establishing the range of costs, we can place some realistic boundaries on discussions of 
resources required for successful, local assessment reform. We need to spend time identifying and 
capturing what the real costs associated with assessment reforms are--whether the cost burden falls upon 
the district, teachers, or students. Understanding these types of costs goes far beyond analysis of entries in 


a ledger. 


In summary, educators and policy makers need to develop more sophisticated models of costs and benefits 
beyond the expenditure analysis most typically presented. Such models must include analyses of 
alternatives both in terms of expected benefits from the chosen option as well as the benefits foregone 
from those rejected. Rarely will reforms be paid fully by incorporating substitution methods. 
Consequently, policy makers must provide evidence of efficiency along with the arguments of alignment 


that are typically proffered when new programs are introduced. 
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